Tactical robot controller

ABSTRACT

An unmanned systems operator control system includes a hand held controller with a set of switches and control enumeration software specially configured to report a superset of virtual switches based on the physical switches. A core unit includes a first unmanned system control application subscribing to a first switch subset of the superset and outputting commands controlling a first unmanned system based on activation of the set of switches. A second unmanned system control application subscribes to a second switch subset of the superset and outputs commands controlling a second unmanned system based on activation of the set of switches. A mode switching subsystem is configured, in a first state, to map the set of switches to the first switch subset and, in a second state, to map the set of switches to the second switch subset.

GOVERNMENT RIGHTS

This invention was made with U.S. Government support under Contract No.#N00178-11-C-1025 awarded by Naval Surface Warfare Center. TheGovernment may have certain rights in aspects of the invention.

FIELD OF THE INVENTION

The subject invention relates to robot controllers.

BACKGROUND OF THE INVENTION

There are a variety of remotely controlled mobile robots available andeach typically include their own specific “Operator Control Unit” (OCU)used by an operator to wirelessly drive and steer the robot, operate itsarm, maneuver its cameras, and the like. That is, the OCU for thewell-known QinetiQ-NA, Inc. “Talon” robot cannot be used as an OCU tocontrol the iRobot Corp. “PackBot” robot and vise-versa. The same issometimes true with respect to different model robots of even the samecompany or vendor and also true with respect to unmanned aerial andwatercraft vehicles, different remote sensors, weapons, and the like.For example, the OCU of the “Raven” drone cannot be used to control the“Talon” robot.

The result is an incentive for a so-called “common controller”. The U.S.Army program is called the “AN/PSW-2 Common Controller”.

U.S. Pat. No. 8,119,109 of iRobot Corp., incorporated herein by thisreference, discloses a “twin grip design” input device connected to arugged laptop computer for controlling teleoperative remote controllablevehicles. The software of the computer is stated to be proprietary butalso includes “common OCU software”. How the signals output by theswitches of the device are processed by the computer is not disclosedbut different button function modes are possible.

Others are developing tactical control units with scalable warfighter-machine interfaces. Still others are attempting to adapt gametype controllers for controlling unmanned aerial and ground vehicles.

Those skilled in the art have also studied controlling multiple robotsor payloads. One example is a soldier controlling a flying drone and, atthe same time, controlling a ground robot using one controller. Thehandheld controller could have one switch set for the ground robot andanother switch set for the drone but then the handheld controller wouldbe large and complex.

U.S. Pat. No. 5,394,168, incorporated herein by this reference,discloses a game controller with buttons for controlling movement of oneobject and an optical subsystem for controlling movement of a differentobject. U.S. Pat. No. 8,199,109 states that the controller has “modechanging software” which is not disclosed. The '109 patent also statesthe controller can be used to control two or more payloads.

SUMMARY OF THE INVENTION

A problem occurs when a “common” OCU includes proprietary hardwareand/or software and the different teleoperatable remotely controlledvehicles each include proprietary hardware and software. Arriving in atrue common OCU which is able to remotely control vehicles of differenttypes and from different vendors is challenging.

In one example of this invention, vendors of different remote controlvehicles can keep their controlling hardware (e.g., radios and the like)and their controlling software proprietary. The different radios caneven transmit at different frequencies, use different protocols, and thelike. Source code of proprietary software is not required. The commoncontroller system of one example of the invention can be used to controlany remotely controlled vehicle because the common controller systemaccepts any remote control vehicle's radio pack and proprietary softwarefor controlling the radio pack.

The invention features, in one example, the ability to use acommercially available handheld controller without the need for modechanging software. The radios and radio software applications fordifferent unmanned system are accommodated. When the controller systemwill be used to control a drone, robot, or the like, its radio andsoftware applications are incorporated.

Featured is an unmanned systems operator control system comprising a setof switches and control enumeration software configured to report asuperset of virtual switches. A first unmanned system controlapplication subscribes to a first switch subset of the superset andoutputs commands controlling a first unmanned system based on activationof the set of switches. A second unmanned system control applicationsubscribes to a second switch subset of the superset and outputscommands controlling a second unmanned system based on activation of theset of switches. A mode switching subsystem is configured, in a firststate, to map the set of switches to the first switch subset and, in asecond state, to map the set of switches to the second switch subset.

In one example, the set of switches are associated with a handheldcontroller and include at least one joystick and a plurality of Xbuttons and the superset includes at least two joysticks and 2X buttons.The mode switching subsystem is also typically associated with thehandheld controller. One handheld controller further includes a firstscreen for the first unmanned system and a second screen for the secondunmanned system. In one version, the first unmanned system controlapplication and the second unmanned system control application areassociated with a core module. A first radio is for transmitting controlcommands to the first unmanned system. A second radio is fortransmitting control commands to the second unmanned system.

The system may also include a video server and transmitter forwirelessly transmitting video displayed on the video screen(s).

Also featured is a method of controlling a plurality of unmanned systemswith one handheld controller having a set of switches. The set ofswitches are mapped to a first switch subset of a superset of virtualswitches. The method includes subscribing to the first switch subsetwhen controlling a first unmanned system using the set of switches. Theset of switches are mapped to a second switch subset of a superset ofvirtual switches and the second switch subset is subscribed to whencontrolling the second unmanned system using the set of switches.Preferably, video associated with the first unmanned system is displayedon a first screen and video associated with the second unmanned systemis displayed on a second video screen.

Also featured is a tactical robot controller for first and second robotscomprising a first unmanned system control application is configured tooutput commands controlling a first unmanned system via a first radio.At least a second unmanned system control application system isconfigured to output commands controlling a second unmanned system via asecond radio. A handheld controller includes a set of switches forcontrolling the first and second unmanned systems and also softwaredelivering switch data to the first unmanned system control applicationand to the second unmanned system control application. A mode selectionis switchable between the first and second unmanned systems. In onepreferred embodiment, the handheld controller includes softwareconfigured to report a superset of virtual switches, the first unmannedsystem control application subscribes to a first switch subset of thesuperset, the second unmanned system control application subscribes to asecond switch subset of the superset, and the handheld controllersoftware is configured to map the set of handheld controller switches tothe first switch subset when the mode selection is switched to the firstunmanned system and to map the set of handheld controller switches tothe second switch subset when the mode selection is switched to thesecond unmanned system.

Also featured is a tactical robot control method for first and secondrobots comprising controlling a first unmanned system by deliveringswitch data to a first unmanned system control application issuingcommands to the first unmanned system via a first radio, controlling asecond unmanned system by delivering switch data to a second unmannedsystem control application issuing commands to the second unmannedsystem via a second radio, and switching between control of the firstunmanned system and the second unmanned system.

The method may further include the step of reporting a superset ofvirtual switches based on a set of physical switches generating theswitch data, subscribing the first unmanned system control applicationto a first switch subset of the virtual superset, subscribing the secondunmanned system control application to a second switch subset of thevirtual superset, and switching between control of the first unmannedsystem and the second unmanned system by mapping the set of physicalswitches to the first switch subset when controlling the first unmannedsystem and mapping the set of physical switches to the second switchsubset when controlling the second unmanned system.

The subject invention, however, in other embodiments, need not achieveall these objectives and the claims hereof should not be limited tostructures or methods capable of achieving these objectives.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

Other objects, features and advantages will occur to those skilled inthe art from the following description of a preferred embodiment and theaccompanying drawings, in which:

FIG. 1 is a highly schematic view showing an example of a handheldcontroller in accordance with the invention;

FIG. 2 is a block diagram showing the primary components associated witha tactical robot controller in accordance with one example of theinvention;

FIG. 3 is a schematic view of the core module shown in block diagramform in FIG. 2; and

FIG. 4 is a flow chart depicting the primary steps associated with amethod in accordance with an example of the invention and theprogramming of the software disclosed herein.

DETAILED DESCRIPTION OF THE INVENTION

Aside from the preferred embodiment or embodiments disclosed below, thisinvention is capable of other embodiments and of being practiced orbeing carried out in various ways. Thus, it is to be understood that theinvention is not limited in its application to the details ofconstruction and the arrangements of components set forth in thefollowing description or illustrated in the drawings. If only oneembodiment is described herein, the claims hereof are not to be limitedto that embodiment. Moreover, the claims hereof are not to be readrestrictively unless there is clear and convincing evidence manifestinga certain exclusion, restriction, or disclaimer.

FIG. 1 depicts an example of a handheld controller 20 with first screen22 a, second screen 22 b, and housing 26 for a set of switchesincluding, in this particular example, joysticks 28 a and 28 b and 20buttons such as button 28 c, 28 d, 28 e and the like. Also featured ismode button 32.

In general, the user can control two or more robots such as ground robot34 depicted on screen 22 a and flying drone 36 as depicted on screen 22b. Video from cameras on the robots is typically delivered to screen 22a and 22 b via the robot's radio set. In one example, in a first mode,joystick 28 a may be used to control (e.g., turn) ground robot 34 while,in a second mode, joystick 28 a is used to control (e.g., turn) drone36. The different modes are activated by mode button 32. In otherexamples, there is only one screen and video from the robot cameras isshown in an alternate fashion on the screen in conjunction with theactivation of mode button 32 and/or by splitting the screen.

One preferred architecture for a tactical robot controller is shown inFIG. 2 where controller 20 includes one or more touch screens 22, a setof switches 28, and processor 40 running control enumeration software 42as well as other software. Processor 40 may be or include one or moremicroprocessors, application specific integrated circuits, programmablelogic devices, controllers and the like. The control enumerationsoftware is configured to deliver switch data to and output a message(via processor 40 and USB cable 44) to core 50 concerning the switch setof handheld controller 20. In this particular example, controlenumeration software 42 would normally generate a USB controlenumeration message indicating handheld controller 20 has two joysticksand 20 buttons.

But, in order to control multiple robots, control enumeration software42 is specifically configured to report a superset of virtual switches.e.g., four joysticks and 40 buttons.

Core 50 includes central processing unit 52 (configured for a windowsenvironment, for example) running application software modules such asfirst unmanned system control application 54 a and second unmannedsystem control application 54 b. These applications, in general,generate control signals for their respective robots based on theactivation of switches associated with handheld controller 20.

In this invention, application 54 a specially subscribes to a firstswitch subset of the aforementioned superset of virtual switches.Application 54 b specially subscribes to a second switch subset of theaforementioned superset of virtual switches. In the example above,application 54 a may subscribe to the first and second joysticks andbuttons 1-20. Application 54 h then subscribes to the third and fourthjoysticks and buttons 21-40.

When mode selection switch 32 is activated to control the first robot,software 42 maps the actual physical set of controller switches to thefirst virtual switch subset, here the first and second joysticks andbuttons 1-20. When the mode selection 32 is activated to control thesecond robot, software 42 maps the actual physical set of switches tothe second virtual switch subset, here the third and forth joysticks andbuttons 21-40.

In this way, application software 54 a which subscribes only to thefirst virtual switch subset controls the first robot 34 (via signalstransmitted by radio 56 a) when the mode for the first robot is selectedand application software 54 a ignores signals from handheld controller20 when the mode for the second robot is selected since now the handheldcontroller has mapped its physical switches to the second virtual switchsubset which application 54 a does not subscribe to. Further,application software 54 b which subscribes only to the second virtualswitch subset, controls the second robot 36 (via signals transmitted byradio 56 b) when the mode for the second robot is selected andapplication software 54 b ignores signals from the handheld controller20 when the mode for the first robot is selected since now the handheldcontroller has mapped the physical switches to the first virtual switchsubset which application 54 b does not subscribe to.

If three robots are to be controlled, the superset of virtual switchescould be larger and there may be three switch subsets and three controlapplications each subscribed to a different subset and transmittingsignals to three different radios. Note that the subsets need not, whencombined, coincide with the superset, there may be overlap between thesubsets, and the superset need not coincide with all the physicalbuttons.

In one particular example, when the mode for control of robot 34 isselected and joystick 28 a, FIG. 1 is toggled to the right, software 42has mapped joystick 28 a to joystick No. 1. Software 42 reports to CPU52 a switch data message that joystick No. 1 has toggled to the rightand application 54 a, which subscribes to joystick No. 1, sends acommand via radio 56 a to turn robot 34 to the right. When the mode forthe control of robot 36 is selected and joystick 28 a, FIG. 1 is toggledto the right, software 42 has mapped joystick 28 a to joystick No. 3.Software 42 reports to CPU 52 a message that joystick No. 3 has toggledto the right and application 54 b, which subscribes to joystick No. 3,sends a command via radio 56 b to turn drone 36 so it banks to theright.

Note how radios 56 a and control software 54 a may be provided from thevendor of robot 34 and radio 56 b and application software 54 b may beprovided from the vendor of robot 36. By loading a vendor software andcarrying out the subscription and mapping steps described above for thephysical switches, a true common controller is realized.

FIG. 3 shows an example of electronic subsystem core module 50physically and electrically coupled, in one particular example, to powersupply module 60 which includes swappable batteries 62 a and 62 b.Weighing less than 5.5 lbs, this combination can be placed in a smallmolle pack or pouch coupled to a soldier's existing molle pack. Core 50includes heat sink 64, a Wi-Fi/Bluetooth antenna, and various ports orconnectors. Handheld controller 20, FIG. 2 is electrically coupled tocore module 50, FIG. 3 via USB cable plugged into connector 66. Variousradio packs such as radio pack 56 a and 56 b, FIG. 2 can be electricallycoupled to core 50, FIG. 3 via a cable plugged into connectors 66 a and66 b. Other ports are for a headset, other radio packs, and/or apersonal computer interface (USB, Ethernet, or the like). Power supplymodule 60 is shown physically as well as electrically coupled to coremodule 50 but this is not a necessary limitation of the invention. Powersupply module 60 can be coupled to core module 50 via a cable, forexample.

A solider using handheld controller 20, FIG. 1 may have core module 50and power supply module 60 located in the bottom half of a molle pouchwhile the radio packs can be located in the upper half of the mollepouch. The molle pouch can be coupled to an assault pack.

As discussed above, by changing the radio pack and executing differentsoftware programs operable on the core module, the solider can use thehandheld controller to control different ground robots, different UAVs,remotely controlled watercraft, various pods of an aircraft, sensors,and the like.

The vendors of such devices typically provide their own radio packsplugged into and software programs loaded onto core module. In order tomake a software program operate correctly, interface control softwaremodules can be included using a USB button map provided to the variousvendors. A vendor can keep its software programs, protocols, and theconfiguration of a radio pack proprietary.

Core module 50 may also include video server 51 electrically connectedto a video out lead of CPU 52 (e.g., a 1-2 GHz dual core processoroperating Windows XP OS and employing 4 GB of RAM) for digitizing to anMPEG video stream the analog video signal processed by CPU 52. This MPEGvideo signal is transmitted by Wi-Fi transmitter 53 so other personnelcan view, on a smart phone, for example, the video feed displayed onscreens 22 a and/or 22 b.

Note that in some embodiments the software 42, 54 a, 54 b and the likecan be combined, distributed in various modules, and/or reside infirmware. Other software is typically associated with the system. Ingeneral, there is software configured, as discussed above, to report tothe CPU or other processor(s) a superset of virtual switches, step 80FIG. 4. One robots control application subscribes to a first switchsubset, step 82 and another robots control application subscribes to asecond switch subset, step 84. For a first mode, the set of physicalswitches are mapped to the first switch subset, step 84 and, for theother mode controlling another robot, the set of physical handheldcontroller switches are mapped to the second switch subset, step 88.

Also note that the functionality of processor 40 and CPU 52 may becombined or reside in circuitry and distributed other than as shown.Microprocessors, controllers, application specific integrated circuits,programmable logic devices and the like may be used. Various signalprocessing and other electronic and electrical circuitry and printedcircuit boards are not shown.

Therefore, although specific features of the invention are shown in somedrawings and not in others, this is for convenience only as each featuremay be combined with any or all of the other features in accordance withthe invention. The words “including”, “comprising”, “having”, and “with”as used herein are to be interpreted broadly and comprehensively and arenot limited to any physical interconnection. Moreover, any embodimentsdisclosed in the subject application are not to be taken as the onlypossible embodiments.

In addition, any amendment presented during the prosecution of thepatent application for this patent is not a disclaimer of any claimelement presented in the application as filed: those skilled in the artcannot reasonably be expected to draft a claim that would literallyencompass all possible equivalents, many equivalents will beunforeseeable at the time of the amendment and are beyond a fairinterpretation of what is to be surrendered (if anything), the rationaleunderlying the amendment may bear no more than a tangential relation tomany equivalents, and/or there are many other reasons the applicant cannot be expected to describe certain insubstantial substitutes for anyclaim element amended.

Other embodiments will occur to those skilled in the art and are withinthe following claims.

What is claimed is:
 1. An unmanned systems operator control systemcomprising: a set of switches; control enumeration software configuredto report a superset of virtual switches; a first unmanned systemcontrol application subscribed to a first switch subset of the supersetto output commands controlling a first unmanned system based onactivation of the set of switches; at least a second unmanned systemcontrol application subscribed to a second switch subset of the supersetto output commands controlling a second unmanned system based onactivation of the set of switches; and a mode switching subsystemconfigured, in a first state, to map the set of switches to the firstswitch subset and, in a second state, to map the set of switches to thesecond switch subset.
 2. The system of claim 1 in which the set ofswitches are associated with a handheld controller.
 3. The system ofclaim 2 in which the set of switches includes at least one joystick anda plurality of X buttons and the superset includes at least twojoysticks and 2X buttons.
 4. The system of claim 1 in which the modeswitching subsystem is associated with a handheld controller.
 5. Thesystem of claim 2 in which the handheld controller further includes afirst screen for the first unmanned system and a second screen for thesecond unmanned system.
 6. The system of claim 1 in which the firstunmanned system control application and the second unmanned systemcontrol application are associated with a core module.
 7. The system ofclaim 1 further including a first radio for transmitting to said firstunmanned system said commands controlling the first unmanned system anda second radio for transmitting to the second unmanned system saidcommands for controlling the second unmanned system.
 8. The system ofclaim 1 further including at least a first video screen.
 9. The systemof claim 8 further including a video server and transmitter forwirelessly transmitting video displayed on the first video screen. 10.The system of claim 8 further including a second video screen.
 11. Amethod of controlling a plurality of unmanned systems with one handheldcontroller having a set of switches, the method comprising: mapping theset of switches to a first switch subset of a superset of virtualswitches; subscribing to the first switch subset when controlling afirst unmanned system using the set of switches; mapping the set ofswitches to a second switch subset of a superset of virtual switches;and subscribing to the second switch subset when controlling the secondunmanned system using the set of switches.
 12. The method of claim 11 inwhich the set of switches include at least one joystick and a pluralityof X buttons and the superset includes at least two joysticks and 2Xbuttons.
 13. The method of claim 11 further including displaying videoassociated with the first unmanned system on a first screen anddisplaying video associated with the second unmanned system on a secondvideo screen.
 14. A tactical robot controller for first and secondrobots comprising: a first unmanned system control applicationconfigured to output commands controlling a first unmanned system via afirst radio; at least a second unmanned system control applicationsystem configured to output commands controlling a second unmannedsystem via a second radio; a handheld controller including a set ofswitches for controlling the first and second unmanned systems andsoftware delivering switch data to the first unmanned system controlapplication and to the second unmanned system control application; and amode selection switchable between the first and second unmanned systems.15. The controller of claim 14 in which the handheld controller includessoftware configured to report a superset of virtual switches.
 16. Thecontroller of claim 15 in which the first unmanned system controlapplication subscribes to a first switch subset of the superset and thesecond unmanned system control application subscribes to a second switchsubset of the superset.
 17. The controller of claim 16 in which thehandheld controller software is configured to map the set of handheldcontroller switches to the first switch subset when the mode selectionis switched to the first unmanned system and to map the set of handheldcontroller switches to the second switch subset when the mode selectionis switched to the second unmanned system.
 18. The controller of claim14 in which the handheld controller further includes a first screen forthe first unmanned system and a second screen for the second unmannedsystem.
 19. The controller of claim 14 in which the first unmannedsystem control application and the second unmanned system controlapplication are associated with a core module.
 20. The controller ofclaim 14 further including a first radio for transmitting to said firstunmanned system said commands controlling the first unmanned system anda second radio for transmitting to the second unmanned system saidcommands for controlling the second unmanned system.
 21. The controllerof claim 14 further including a video server and transmitter forwirelessly transmitting video.
 22. A tactical robot control method forfirst and second robots comprising: controlling a first unmanned systemby delivering switch data to a first unmanned system control applicationissuing commands to the first unmanned system via a first radio;controlling a second unmanned system by delivering switch data to asecond unmanned system control application issuing commands to thesecond unmanned system via a second radio; and switching between controlof the first unmanned system and the second unmanned system.
 23. Themethod of claim 22 further including the step of reporting a superset ofvirtual switches based on a set of physical switches generating saidswitch data.
 24. The method of claim 23 further including the step ofsubscribing the first unmanned system control application to a firstswitch subset of the virtual superset and subscribing the secondunmanned system control application to a second switch subset of thevirtual superset.
 25. The method of claim 24 in which switching betweencontrol of the first unmanned system and the second unmanned systemincludes mapping the set of physical switches to the first switch subsetwhen controlling the first unmanned system and mapping the set ofphysical switches to the second switch subset when controlling thesecond unmanned system.